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The findings from this review do not reflect the full body of research evidence on InsideTrack. 


What is this study about? 

The study examined whether InsideTrack, a per- 
sonalized student coaching service for college stu- 
dents, increased rates of staying in and graduating 
from college. 

The study analyzed data on about 13,500 students 
who were enrolled in one of eight higher education 
institutions during the 2003-04 and 2007-08 
academic years. These institutions provided lists 
of students for InsideTrack to include in the study. 

InsideTrack created lotteries that randomly assigned 
groups of students either to receive coaching ser- 
vices from InsideTrack or to serve as the compari- 
son group. Students were moved between groups 
after random assignment in 10 of the 17 lotteries. 

The authors presented two sets of analyses: one 
based on the subset of seven well-executed lot- 
teries (where students were not moved between 
groups after random assignment) and the other 
based on the full set of 17 lotteries. 

The study examined whether students stayed in 
or completed college by comparing the outcomes 
of all students who were randomly selected to 
receive InsideTrack with the outcomes of students 
who were not. 


Features of InsideTrack 


InsideTrack is a provider of one-on-one student 
coaching for college students. It operates 
independently in cooperation with partner 
institutions. 

Coaches assess students’ lives inside and outside 
of school and help them overcome barriers to 
academic success. They contact their students 
regularly and, when possible, use information on 
students’ performance and participation in class 
to inform their discussions. 

Coaching services typically last for two semesters, 
and student participation is voluntary. 


Staying in college was measured at six and 12 
months after randomization for students in all 17 
lotteries. Twelve lotteries also provided informa- 
tion on staying in college for students at 18 and 24 
months. Graduation from college was measured for 
students in three well-executed lotteries. 
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What did the study find? 

For the seven well-executed lotteries, the study found 
that students assigned to receive InsideTrack were 
significantly more likely than students in the compari- 
son group to remain enrolled at their institutions. Six 
months after random assignment, 81 % of students 
in the intervention group were still enrolled, compared 
to 77% of students in the comparison group. After 
12 months, 66% and 51% of the intervention and 
comparison groups, respectively, were enrolled, and 
44% and 37% were enrolled after 18 months. 

There was no significant difference in enrollment rates 
after 24 months. There was also no significant differ- 
ence in completion rates within four years, a result 
based on a subset of three well-executed lotteries. 

For all 17 lotteries, the study found that students 
assigned to receive Inside Track were significantly 
more likely to remain enrolled at their institutions 
than students in the comparison group. Six months 
after random assignment, 63% of students in the 
intervention group were still enrolled, compared with 
58% of those in the comparison group. After 12 
months, enrollment was 49% and 44%, respectively. 
After 18 months, the numbers were 33% and 29%, 
and after 24 months, they were 28% and 24%. The 
study did not examine completion rates within four 
years for all lotteries. 


WWC Rating 


The research on the subset of 
seven well-executed lotteries 
described in this report meets 
WWC evidence standards 
without reservations 

Strengths: The lotteries in this subset are well- 
executed randomized controlled trials with low 
attrition. 

The research for all lotteries 
described in this report meets 
WWC evidence standards 
with reservations 

Cautions: The full set of lotteries includes those in 
which students were moved between groups after 
random assignment. These nonrandomly formed 
groups were equivalent at baseline, so the study 
meets standards with reservations. 
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Appendix A: Study details 

Bettinger, E. P., & Baker, R. (2011). The effects of student coaching in college: An evaluation of a 
randomized experiment in student mentoring (Working Paper No. 16881). Retrieved from: 
http://www.nber.org/papers/w16881. 


Setting 

The study was conducted in eight participating universities during the 2003-04 and 2007-08 
school years. 

Study sample 

Data came from students entering the 2003-04 (five lotteries) and 2007-08 school years (12 
lotteries). Students were college students enrolled in public, private, and proprietary universities. 
Each institution had its own eligibility criteria and provided a list of potential students for Inside- 
Track to randomly assign into two groups. Most institutions provided a representative sample of 
new college students, including many students who were not traditional college age, but some 
schools focused on other subgroups, including full-time students, part-time students, upper- 
classmen, and athletes. InsideTrack then performed two types of randomization: 

(1) For institutions that wanted equally sized groups (seven out of 17 lotteries, referred to as 
“well-executed” lotteries), InsideTrack created two randomly assigned groups of approximately 
equal size, and the institution decided which of the two groups would receive the intervention 
through a coin flip. Following the coin flip, the institution was notified which students were in 
each group. InsideTrack monitored the randomization to make sure that the two groups were 
balanced across observable characteristics. In some cases, students were moved between 
groups to achieve balance before the groups were randomly assigned to the intervention and 
comparison conditions. The authors presented the results for this subset of seven well-executed 
lotteries separately. In these lotteries, 1,768 students were assigned to the intervention group 
and 1 ,768 were assigned to the comparison group. 

(2) For institutions that wanted a smaller comparison group (10 out of 17 lotteries), the institu- 
tion provided InsideTrack with a predetermined size for the comparison group, and InsideTrack 
then randomly assigned two groups to meet those size restrictions. In some cases, students 
were moved between groups to achieve balance after the groups were randomly assigned to 
the intervention and comparison conditions. 

Altogether, 8,049 students were assigned to the intervention group, and 5,506 students were 
assigned to the comparison group. In the overall sample, the average age of students was 31 , 
and about 51 % of the students were male. 

Intervention 

group 

Students in the intervention group received individualized coaching from an InsideTrack coach. 
A coach typically worked with a student for two semesters. Significant time was spent assess- 
ing students’ lives outside of school in such areas as personal time commitments, primary 
caregiving responsibilities, and financial obligations. 
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Comparison 

group 

Outcomes and 
measurement 


Reason for 
review 


The comparison condition received no individualized coaching through InsideTrack. All students 
had access to traditional resources provided through their institutions. 

The primary outcomes were staying in college and completing a degree within four years. 
Students in all 17 lotteries were assessed for staying in college at six and 12 months after 
randomization, and students in 12 lotteries were additionally measured as staying in college 
at 1 8 and 24 months after randomization. Degree completion within four years was measured 
for students in three lotteries, which were part of the subset of seven well-executed lotteries. 
For a more detailed description of these outcome measures, see Appendix B. 

This study was eligible for a single study review by receiving significant media attention. 


August 2012 


Page 4 


WWC Single Study Review 


Appendix B: Outcome measures for each domain 


Staying in school 


Staying in school 


Enrollment (measured 6, 12, 18, 
and 24 months after randomization) 

Enrollment is measured as a binary variable with a value of one if a student is on a list of enrolled students 
provided by a participating institution at a point in time. All institutions provided lists of enrolled students at 
four times after groups were randomized to receive student coaching or the comparison condition: after six 
and 12 months for all lotteries, and after 18 and 24 months for 12 lotteries. 

Completing school 


Completing a degree within four years 

Completing a degree within four years is measured as a binary variable with a value of one if a student 
completes a certificate, an associate's degree, or a bachelor’s degree. Three lotteries within the subset 
of seven well-executed lotteries had information on degree completion within four years. 


Completing school 
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Appendix C: Study findings for each domain 


Mean 

(standard deviation) WWC calculations 


Domain and outcome 
measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Staying in school 

Enrolled 12 months after 
randomization 

Well- 

executed 

lotteries 

7 lotteries/ 
3,527 
students 

0.66 

(0.47) 

0.61 

(0.49) 

0.05 

0.10 

+4 

<0.01 

Domain average for staying in school 





O.IO 

+4 

Statistically 

significant 

Completing school 

Completed a degree within 
four years of the start of 
intervention 

Well- 

executed 

lotteries 

3 lotteries/ 
1,346 
students 

0.35 

(0.48) 

0.31 

(0.46) 

0.04 

0.08 

+3 

<0.10 

Domain average for completing school 





0.08 

+3 

Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) 
in an average student’s outcome that can be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. The table presents results in the staying in school domain from the 
strongest design presented in the study — the design based on the seven well-executed lotteries — measured immediately upon conclusion of the intervention. Later follow-up 
periods for enrollment outcomes are based on smaller samples. Results for completing school are also from the strongest design, though only three of the seven well-executed 
lotteries had information on this outcome. The study is characterized as having a statistically significant positive effect on staying in school because univariate statistical tests are 
reported for each outcome measure, the effect for at least one measure within the domain is positive and statistically significant, and no effects are negative and statistically sig- 
nificant. The study is characterized as having an indeterminate effect on completing school because the mean effect is neither statistically significant nor substantively important. 
Study Notes: The p-values presented here were reported in the original study. All reported results are rounded to two decimal points. 
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Appendix D: Supplemental findings by domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Staying in school 

Enrolled 6 months after 
randomization 

Weil- 

executed 

lotteries 

7 lotteries/ 
3,527 
students 

0.81 

(0.40) 

0.77 

(0.42) 

0.04 

0.09 

+4 

<0.01 

Enrolled 18 months after 
randomization 

Well- 

executed 

lotteries 

3 lotteries/ 
1,344 
students 

0.44 

(0.50) 

0.37 

(0.48) 

0.07 

0.14 

+6 

<0.01 

Enrolled 24 months after 
randomization 

Well- 

executed 

lotteries 

3 lotteries/ 
1,348 
students 

0.38 

(0.48) 

0.35 

(0.48) 

0.03 

0.06 

+2 

>0.10 

Staying in school 

Enrolled 6 months after 
randomization 

All lotteries 

17 lotteries/ 
13,552 
students 

0.63 

(0.48) 

0.58 

(0.49) 

0.05 

0.10 

+4 

<0.01 

Enrolled 12 months after 
randomization 

All lotteries 

17 lotteries/ 
13,553 
students 

0.49 

(0.50) 

0.44 

(0.50) 

0.05 

0.10 

+4 

<0.01 

Enrolled 18 months after 
randomization 

All lotteries 

12 lotteries/ 
11,149 
students 

0.33 

(0.47) 

0.29 

(0.45) 

0.04 

0.09 

+4 

<0.01 

Enrolled 24 months after 
randomization 

All lotteries 

12 lotteries/ 
11,153 
students 

0.28 

(0.45) 

0.24 

(0.43) 

0.03 

0.08 

+3 

<0.01 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) 
in an average student’s outcome that can be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. 

Study Notes: The p-values presented here were reported in the original study. All reported results are rounded to two decimal points. 


August 2012 


Page 7 



WWC Single Study Review 


Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information from requests to the 
authors]) to assess whether the study’s design meets WWC evidence standards. The review reports the WWC’s assessment of 
whether the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting 
evidence on effectiveness. The WWC rating applies only to the summarized results, and not necessarily to all results presented in the 
study. This study was reviewed using the Dropout Prevention review protocol, version 2.0. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2012, August). 

WWC review of the report: The effects of student coaching in college: An evaluation of a randomized experi- 
ment in student mentoring. Retrieved from http://whatworks.ed.gov. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% Ip < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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